DOCUMENT RESUME 



ED 420 699 

AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 
ABSTRACT 

education and industry interests to better meet the needs of students, 
workers, and employers. After the passage of the Perkins Vocational and 
Applied Technology Act in 1990 and its reauthorization in 1992, the U.S. 
Departments of Labor and Education jointly initiated funding for 22 projects 
to develop industry skill standards in diverse industries. The National Skill 
Standards and Assessment Collaborative (NSSAC) has combined some of the 
efforts of the 22 projects to develop cross- industry standards linked to the 
industry skill standards for 4 partner industries, and they have pilot tested 
assessment methods that apply across these industries. This paper reports on 
the cross- industry analysis and subsequent assessment development work with 
one assessment method. First, the paper discusses the individual industry 
standards and their structure, highlighted by an example. Following 
discussion of the individual standard sets, the paper discusses the 
methodology and findings from the cross-industry analysis. To highlight the 
use of cross- industry standards for the purposes of assessment development, 
preliminary efforts in one area, written scenarios, are discussed. The four 
industries are: (1) retail and professional sales; (2) community support 

service worker; (3) health care services; and (4) electronics. The 
Secretary's Commission on Achieving Necessary Skills (SCANS) skills and 
competencies were used as an external referent to help establish a basis for 
the analysis. While portfolios and scenarios were chosen as two 
performance-based assessment methods on which to focus assessment development 
efforts, this paper concentrates on scenarios. The written scenario involves 
presenting a work-related situation for which the respondent is asked to 
develop and present a solution. Developing scenarios for the four industries 
is discussed. (Contains 8 tables, 4 figures, and 14 references.) ( SLD) 



TM 028 374 

Schwager, Mahna T. 

Using Skill Standards To Generate Cross- Industry Areas for 
Certification and Assessment. 

1998-04-00 

3 7p . ; Paper presented at the Annual Meeting of the American 
Educational Research Association (San Diego, CA, April 
13-17, 1998) . 

Reports - Descriptive (141) -- Speeches/Meeting Papers (150) 

MF01/PC02 Plus Postage. 

*Certif ication; Educational Assessment; Evaluation Methods; 
Federal Legislation; *Industry; Licensing Examinations 
(Professions) ; *Performance Based Assessment; *Skills; 
*Standards; Tables (Data) 

*Scenario Writing; Scenarios 

Recently there has been legislation aimed at aligning 



★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★★******************************************** 



* Reproductions supplied by EDRS are the best that can be made * 

* from the original document. * 

******************************************************************************** 




TM028374 



Using Skill Standards to Generate Cross-Industry Areas for 

Certification and Assessment 



Mahna T. Schwager 
WestEd 

April 1998 



Paper presented at the ann ual meeting of the American Educational Research Association 

at San Diego, CA 




PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 
• HAS BEEN GRANTED BY 

yvuwa. 



TO THE EDUCATIONAL RESOURCES | 
INFORMATION CENTER (ERIC) J 



US. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

OThis document has been reproduced as 
received from the person or organization 
originating it. 

Q Minor changes have been made to 

Imnrnwa ron rriHi irfi/MY nilfltifv 



• Points of view or opinions stated in this 
document do not necessarily represent 
official OER I position or policy. y 



2 



Introduction 



For many years, both in education and industry, professionals have relied on standards to help 
determine what students or workers should know and be able to do. Still, the visibility of the 
standards movement in America has increased significantly in the last fifteen years, largely 
because of concerns about the effectiveness of schools (Nation at Risk, 1983) and the 
competitiveness of American industries in the global marketplace (America’s Choice: High 
Skills or Low Wages, 1990). These concerns have led people in both arenas to try to identify and 
define new ways to increase the quality of performance. 

Recently, a wave of legislation has been enacted aimed at aligning education and industry 
interests to better meet the needs of students, workers, and employers. The Carl D. Perkins 
Vocational and Applied Technology Education Act (Perkins II) in 1990 was pivotal in initiating 
the process of broad-based consensus building between education and industry. This law required 
vocational education programs to develop and implement a system of performance standards, 
assessment measures, and services that provide “strong experience in and understanding of all 
aspects of the industry students are preparing to enter....” (Perkins II). 

Following Perkins II in 1992, the U. S. Departments of Education and Labor jointly initiated 
funding for 22 projects to develop industry skill standards in diverse industries. Three 
interlocking pieces of legislation in subsequent years further strengthened the call for integrated 
education and industry standards: the School-to-Work Opportunities Act (STWOA), Goals 2000: 
Educate America Act, Improving America’s Schools Act, and National Skill Standards Act, 
which establishes the National Skill Standards Board (NSSB). Taken together, these legislative 
mandates promote the development of voluntary systems of national academic and industry skill 
standards and assessments. Presently, the NSSB is funding “voluntary partnerships” with broad 
member representation to endorse skill standards in different economic sectors. 



The National Skill Standards and Assessment Collaborative (NSSAC) 



The National Skill Standards and Assessment Collaborative (NSSAC) combined the efforts of 
four of the original 22 national skill standards pilot projects to address issues concerning 
assessment and certification. With WestEd as the lead partner, NSSAC included four diverse 
industries — health care (represented by WestEd), electronics (American Electronics 
Association, AEA), human services (Human Services Research Institute, HSRI), and retail 
(National Retail Federation, NRF). With support from the U. S. Department of Labor (DOL) and 
the National Skill Standards Board (NSSB), NSSAC has (1) developed cross-industry standard 
areas or categories that are linked to the industry skill standards developed for the four partner 
industries, and (2) pilot-tested assessment methods that apply across these industries, based on 
these standards (see WestEd, 1998 for a complete discussion of this project and the assessment 
development and piloting). 




A central expectation of the skill standards “community” is that they will be able to identify the 
common foundation of knowledge and skills needed across the majority of occupations within the 
economy (Hoppe & Wills, 1996). A major focus of the NSSAC project was to identify common 
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standard areas across the four participating industries that could be used as a basis for assessment 
development. These standards are assumed to be essential for industry, applicable to secondary and 
post-secondary education, and linked to a high level of challenge and expectation. 

To proceed in identification of common standards, WestEd staff first examined each individual 
industry’s skill standards; then we looked for commonalities or general skills described across 
all sets of standards. 

Because of differences across the targeted sets of standards in the level of generality vs. detail 
and industry-specific language, we used the Secretary's Commission on Achieving Necessary 
Skills (SCANS, 1992) skills and competencies as an external referent to help establish a basis for 
analysis. SCANS identifies general competencies and a foundation of basic skills, thinking 
skills, and personal qualities necessary for employment. The SCANS report describes five 
competencies (Resources, Interpersonal, Information, Systems, and Technology) and a three-part 
foundation of skills and personal qualities (Basic Skills, Thinking Skills, and Personal Qualities). 
These eight components are further divided into sub-skills which are integrated in various ways 
for performance of workplace tasks. SCANS represents a workplace foundation and provides a 
good reference point for pinpointing overlap or commonalties. 

This paper, which draws on the Cross-Industry Assessment and Certification: Framework and 
Implementation Guide (WestEd, 1998), reports on the cross-industry analysis and subsequent 
assessment development work with one assessment method. First, the paper discusses the 
individual industry standards and their structure, highlighted by an example. Following 
discussion of individual standard sets, the paper discusses the methodology and findings from 
our cross-industry analysis. To highlight the use of cross-industry standards for the purposes of 
assessment development, preliminary efforts in one area, written scenarios, are discussed. 



National Retail Federation - Retail and Professional Sales 

The National Retail Federation (NRF) Skill Standards focus on one occupational area. 
Professional Sales Associate. The retail industry employs a large proportion of the entry-level 
workforce. Retail workers are often younger and less likely to have college degrees than workers 
in other industries. Traditionally, retail work tends to involve part-time jobs (Bailey & 

Bernhardt, 1996) but provides opportunities to learn the types of customer service and sales skills 
that are useful in many industries. Professional Sales Associates represents the majority of entry- 
level jobs. 

The skill standards are accompanied by an implementation guide and suggested performance 
indicators for assessment. The indicators are intended to be tailored for different work situations. 
The implementation guide includes some suggested assessment techniques. 

The NRF used a DACUM (Developing a Curriculum) procedure to develop the skill standards. 

A DACUM procedure is one type of systematic occupational analysis that allows for 
identification of job tasks. Groups of retail sales associates and managers participated in this 
process. National groups, including high-performance workplaces, validated the tasks and skills, 
ranked their importance, and identified performance indicators. 



Structure of the Standards 



The NRF Skill Standards consist of six skill modules {Provide Personalized Customer Service, 
Sell and Promote Products, Monitor Inventory, Maintain Appearance of Department/Store, 
Protect Company Assets, and Work as Part of a Department/Store Team). The modules offer a 
flexible framework for standards use, as some modules may be more relevant to some sales 
situations than others. From two to three elements or key features comprise each skill module. 
From three to 1 1 key tasks or skills comprise each element. The key tasks include performance 
indicators or explicit examples of behaviors involved in successfully performing the task. 

For example. Work as Part of a Department/Store Team is a skill module containing two 
elements. Support Co-Workers and Create Competitive Advantage. As shown in Figure 1, one 
element. Support Co-Workers, is comprised of five key tasks. Competent performance of the 
first key task {Share ideas and information about selling, marketing, and products) might be 
documented through two suggested performance indicators: Share information fully and in a 
timely manner and Contribute experience and knowledge of products with fellow associates. 
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Figure 1. 

Structure of National Retail Federation Skill Standards 
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Human Services Research Institute — Human Services 



The Community Support Skill Standards also focus on a single occupational area, the Direct 
Service Worker. Institution-based service workers are excluded; the standards apply primarily 
for community-based service delivery contexts. Human services delivery contexts are gradually 
shifting from institutional and centralized environments to community-based environments. A 
majority of direct service workers now work in community -based organizations and the skill set 
needed in a community context is sufficiently different from skills needed in institutional settings 
that they must be treated differently. The Community Support Skill Standards reflect industry 
direction and a philosophical shift toward community-based services. 

The standard-development process included a job analysis and DACUM process, as well as 
validation by review of stakeholders. Master workers, identified by level of experience and 
supervisor recommendation, among other factors, participated in a series of workshops to define 
the major competencies, activities, knowledge and skills involved in their jobs. Results of the 
workshops were validated by a national survey. A Standards Development Team wrote the 
standards document based on the job analysis. Stakeholder groups, including clients, families, 
providers, workers, teachers, trainers, and technical committee reviewed draft documents and 
made recommendations for revisions. 



Structure of the Standards 

The Community Support Skill Standards are organized into twelve standard units. Each unit 
represents a broad, functional theme of human service work, called a competency area. Specific 
functions for each area are broken into two to five skill standards. Each skill standard is linked 
to one or more sample activities that are relevant to and representative of what human service 
workers actually do. Each sample activity is followed by one or more performance indicators 
which provide a basis for measuring performance. The performance indicators consist of 
observable worker behaviors, client reports, and worker self-reports. 

An example of a skill standard in the HSRI competency area. Organizational Participation , 
which involves participation as a member of a team, is displayed in Figure 2. 
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Figure 2. 

Structure of Human Services Research Institute Skill Standards 
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WestEd — Health Care Services 



The National Health Care Skill Standards Project (NHCSSP) standards focus on occupational 
clusters or sets of related occupations and functions. A regional educational research laboratory, 
WestEd, formerly Far West Laboratory, directed the development process, in collaboration with 
numerous health care industry, education, and labor organizations. Representatives of as many 
aspects of health care as possible participated in the development process to provide for a variety 
of perspectives. The health care industry is well-organized with multiple levels of existing 
certifications (Grubb, 1996) and the standards provide a broad base within this industry context. 
The standards are designed to apply primarily to entry and technical level workers. 

Standards development began with review of existing task analyses of health care occupations. 
Broad categories of skills were identified and synthesized into task summaries. Committees 
formed of representatives from health, labor, and education drafted standards based on the task 
summaries and committee members’ shared experience and expertise. Validation proceeded in a 
three-pronged effort, including external review by a range of participants not involved in 
generating the standards, survey of a targeted group of health care experts and practitioners, and 
focus groups of workers and supervisors at a range of delivery sites across the country. In 
addition, responses to a brief questionnaire at the end of the standards document were examined. 
Twelve industry and education sites piloted various applications of the standards. 



Structure of the Standards 

The NHC Skill Standards are organized into a set of eight core skills and four occupational 
clusters: Therapeutic, Diagnostic, Information Services, and Environmental. The eight core 
skills apply to the full range of health care workers; standards in each cluster apply to jobs that 
fall into that cluster. Two clusters. Therapeutic and Diagnostic, have a common core of five 
standards that apply to both sets of related occupations. 

The NHC Skill Standards are content standards only; the pilot studies were intended to inform 
development of performance standards. Each standard includes a brief title describing the skill, a 
more detailed description of the knowledge and skills required for attainment of the standard, and 
four to seven specific examples for illustration. In addition to the eight core skill standards, there 
are 3 1 standards across the four occupational clusters. 

For example, one of the common core standards is Teamwork. This standard appears in the 
National Health Care Skill Standards handbook as follows: 
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TEAMWORK 

Health care workers will understand the role and responsibilities of individual members as 
part of the health care team, including their ability to promote the delivery of quality 
health care. They will interact effectively and sensitively with all members of the health 
care team. 

The following may be included: 

• Practice team membership skills, such as cooperation, leadership, and anticipation of the 
needs of coworkers 

• Respect cultural and religious differences of team members 

• Interact with others consistent with the health care team structure and lines of authority 

• Manage conflict within the workplace through consideration of others’ points of view 

• Respect interdisciplinary differences among team members 



American Electronics Association ~ Electronics 

The American Electronics Association (AEA) Skill Standards consists of five components, four 
sets of skill standards for different occupational areas and a framework for foundation skills and 
workplace competencies. The foundation skills and competencies are derived from SCANS. 
AEA supplemented the SCANS categories and tailored SCANS definitions, with changes related 
primarily to technological and business-specific skills and competencies. 

The AEA Skill Standards represent four different occupational areas. Each occupational area 
consists of five to seven critical functions or “what one must be able to do to achieve the key 
purpose” of that area. Two to five activities define each critical function; each activity has 
associated performance indicators. The performance indicators offer ways of assessing whether 
the activity has been performed competently. Each activity has at least three and, some, as many 
as nine associated performance indicators. According to the AEA standards handbook, Setting 
the Standard: A Handbook on Skill Standards for the High-Tech Industry , each critical function 
can be thought of as a competency module. The system is designed to enhance flexibility so that 
demonstrating competency in a critical function in one occupational area is likely to be 
transferable to other occupations and fields. The standards are designed to be applied primarily 
to non-baccalaureate level workers. 

The standard development process began with occupational analysis groups that included 
employees of over 100 member companies. During these sessions, participants identified the key 
purpose for each of the four occupational areas and collaboratively drafted the skill standards. A 
survey of key stakeholders validated the standards. Over 2000 workers and supervisors from a 
cross-section of high-tech companies, ranging from software to aerospace electronics, responded. 
Stakeholders rated the importance of each activity to doing their job and the importance of each 
performance indicator on a scale of “1” or “Of Little Importance” to “4” or “Very Important.” 
Ninety-nine to 100% of all activities and performance indicators received an average rating of at 
least 2.75, or better than “Somewhat Important.” No activities received the lowest rating. 
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Generalized AEA Occupational Area 



The AEA skill standards presented an unique problem in identifying commonalities across the 
standard sets as four occupational areas are defined. Examination of the skill standards for the 
four occupational areas revealed similarities between the skill sets for three of the occupational 
areas and marked differences with the skill set of the fourth. The fourth skill standard set, for 
Manufacturing Specialist Team Leader, reflected line management duties and responsibilities, 
particularly reflective of a coach-manager role. Nearly all of the key activities listed for this 
occupational area (18 of 20 activities) began with “Ensure team” or “Help team.” Such 
terminology indicates a leadership rather than frontline worker role. Skill standards for the other 
three occupational areas included key activities at the non-managerial level. 

For the purposes of analysis, we collapsed the three non-managerial occupational areas into one 
generalized occupational area.- The critical functions for each of these occupational areas were 
mapped to broader functions, which we then used as the basis for comparison to SCANS. This 
approach allowed us to retain functions unique to the industry but produced a generalized set of 
functions and activities that could be compared to SCANS and the skill standards of the three 
other industries. As noted in the AEA standards handbook, some critical functions are very 
similar across electronics occupations, as well as across industries. The skill standards for 
Manufacturing Specialist Team Leader, not representing entry-level skills, was not included in 
the analysis. The general occupational model consisted of seven critical functions reorganized in 
Table 1 as follows: 1 

Table 1. 

General Occupational Model Used for AEA Skill Standards Crosswalk 



Critical Functions for Each Occupational Area in Relation to Generalized Model 


Generalized Occupational 
Area 


Manufacturing Specialist 


Administrative f 
Information Services 
Support 


Pre/Post Sales 


Function One: 
Customer Needs 


Establish customer needs 


Identify customer needs and 
plan work activities 


Identify customer and 
customer needs 


Function Two: 
Communication 


Initiate and sustain 
communication processes and 
procedures 


Initiate and facilitate 
communication 




Function Three: 
Continuous Improvement 


Ensure production process 
meets business requirements 




Promote improved customer 
service and satisfaction 


Function Four: 

Design and Develop Work 
Processes and Procedures 


Determine design workability 
and manufactorability 


Develop, implement, and 
evaluate work processes and 
procedures 


Propose solution that satisfies 
customer needs 


Function Five: 
Resource Management 


Use human resources to 
manage work flow 


Manage resources 


Facilitate implementation and 
delivery of solution 



( Continued next page) 




1 Note that the different AEA occupational areas listed distinctly different performance indicators for the same 
critical functions. Performance indicators are at a more job-specific level than the activities listed above and reflect 
important differences within industry and occupational areas. 
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Table 1. ( Continued ) 

General Occupational Model Used for AEA Skill Standards Crosswalk 



Critical Functions for Each Occupational Area in Relation to Generalized Model 


Generalized Occupational 
Area 


Manufacturing Specialist 


Administrative/ 
Information Services 
Support 


Pre/Post Sales 


Function Six: 
Planning Strategies to 
Accomplish Work 


Select, obtain, and optimize 
available machines and 
equipment to meet product 
process requirements 


Manage schedules, activities, 
and events to achieve 
objectives 


Develop options to meet 
customer needs 


Function Seven: 
Information Tracking and 
Handling 


Make products that meet 
customer specifications 


Generate and maintain 
documents and information 





Figure 3, on the next page, graphically displays Function Two: Communication from the 
generalized model, similarly to our examples from the other skill standards sets. 
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Cross-Industry Analysis 



Comparison of the four standard sets can be accomplished in two ways. First, commonalities 
among the four industry standards can be identified and a core set of cross-industry standard 
areas specified. Because of differences in the level of generality vs. detail and use of industry- 
specific language, this is a difficult task. 

Alternatively, a well-defined and validated set of general workplace skill and competencies can 
be used as a point of reference in thinking about the different standards sets to help generate a 
broad set of competency areas that applies across industries. This is the approach we chose, 
using SCANS skills and competencies as our reference point for crosswalking the industry skill 
standards. 



Methodology 

To proceed with the analysis, we constructed a matrix and crosswalked the SCANS foundation 
skills and competencies and the skill standards from each of our industry partners, retail and 
professional sales, human services, electronics, and health care services. Initially, we assigned a 
code to each SCANS sub-skill or sub-competency. For example, we assigned the code FI to the 
SCANS Foundation Skill, Reading. Table 2 lists SCANS foundation skills and workplace 
competencies and their assigned codes. 



Table 2. 

SCANS Foundation Skills and Competencies 



SCANS Foundation Skills 


SCANS Competencies 


Basic Skills 


Resources 


• Reading (FI) 


• Allocates Time (Cl) 


• Writing (F2) 


• Allocates Money (C2) 


• Arithmetic (F3) 


• Allocates Materials and Facility Resources (C3) 


• Mathematics (F4) 


• Allocates Human Resources (C4) 


• Listening (F5) 

• Speaking (F6) 


Interpersonal 


Thinking Skills 


• Participates as a Member of a Team (C5) 

• Teaches Others (C6) 


• Creative Thinking (F7) 


• Serves Clients/Customers (C7) 


• Decision Making (F8) 


• Exercises Leadership (C8) 


• Problem Solving (F9) 


• Negotiates (C9) 


• Seeing Things in the Mind’s Eye (F10) 


• Works with Cultural Diversity (CIO) 


• Knowing How to Learn (FI 1) 

• Reasoning (FI 2) 


Information 


Personal Qualities 


• Acquires and Evaluates Information (Cl I) 

• Organizes and Maintains Information (02) 


• Responsibility (F13) 


• Interprets and Communicates Information (03) 


• Self-Esteem (F14) 


• Uses Computers to Process Information (04) 


• Sociability (F15) 

• Self-Management (FI 6) 


Systems 


• Integrity /Honesty (FI 7) 


• Understands Systems (05) 




• Monitors and Corrects Performance (06) 

• Improves and Designs Systems (07) 

Technology 

• Selects Technology (08) 

• Applies Technology to Task (09) 

• Maintains and Troubleshoots Technology (C20) 
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For each standard, the point of overlap between each SCANS component and the industry skill 
standard was evaluated. All of the SCANS sub-skills reflected in that standard were coded in the 
appropriate box of that industry’s matrix. We coded one matrix for each set of industry skill 
standards in this manner. For example, Table 3 displays the National Health Care Skill 
Standards matrix as coded in relation to the SCANS sub-skills and competencies. When a sub- 
skill or competency was deemed necessary for accomplishing the standard, although not always 
explicitly stated, the coder inferred that it was necessary and applied the code. Thus, the coding 
represents a generous interpretation of the necessary foundation skills underlying each standard. 
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Matrix Crosswalking the National Health Care Skill Standards and SCANS Foundation Skills and Competencies 
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Two coders independently coded one set of skill standards, the health care industry standards. 
Intercoder reliability was computed 2 (reliability = 66% interrater agreement). Discussions 
between coders revealed differences related to interpretation of SCANS thinking skills, problem 
solving, and decision making. One rater tended to consider behaviors involving the use of logic 
and application of rules as problem solving while the other rater coded such behaviors as 
decision making. A second interrater difference related to the hierarchical nature of some 
SCANS competencies, with one rater listing all competencies in a component, while the other 
listed only the highest level competency, resulting in fewer overall matches. All other sets of 
industry skill standards were similarly coded by a single rater. In the case of the electronics 
industry, we used the synthesized general occupational standards. 

To create our cross-industry categories, we first summed the occurrence of each individual code 
over the standards for each industry. In this way, we were able to describe the most common 
sub-skills reflected in a set of standards, as well as any clusters of sub-skills and competencies 
that repeated across the standards for that industry. 



Results 

In terms of the SCANS competencies the following can be noted across the four sets of industry 

standards (see Table 4 for results for each industry standard set): 

• Information competencies are common and the most prevalent SCANS competencies across 
all sets of standards. 

• Interpersonal competency areas are common and, in the retail, human services, and health 
care industry standards, as prevalent as Information competencies. In the electronics 
industry, Resources competencies were more prevalent than Interpersonal. 

• Understanding and working effectively within systems is a common SCANS competency. 
Systems, itself, was identified in the health care skill standards as a separate standard area. 

• Apply Technology to Task or using technology (sometimes coded with Using Computers to 
Process Information) is a competency common across all four industries in SCANS 
technology area. 

Findings regarding the SCANS foundation skills follow. 

• Decision making, problem solving, and reasoning are SCANS thinking skills which are 
necessary across industries and often are embedded in the same standard. 

• Responsibility and, to a lesser extent, sociability are SCANS personal qualities common 
across these industries. 




2 Intercoder agreement was computed using the following formula: reliability = number of agreements/total 
number of agreements + disagreements. According to Miles and Huberman (1994), 70% intercoder agreement for 
initial use of conceptual coding systems is recommended. However, interrater reliabilitiy can be expected to 
improve with discussion of interrater differences and recoding. 
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• Mathematics is related to fewer standards than the other SCANS foundation skills, but is 
necessary for meeting the standard where involved. 

• In the retail, human services, and health care industries, oral literacy skills, listening and 
speaking, are the most needed or minimum foundation skills, followed by reading. The 
standards for the three service industries focus on communicating with clients and customers, 
eliciting verbal and written information from clients, and sharing information with clients 
that the worker has gathered from different sources. 

• Written literacy skills seemed to be the most prevalent basic skill in the electronics industry 
standards. The standards focused on documenting and tracking a variety of different 
processes and developing plans and strategies 
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Table 4. 

Number of Standards Involving Different SCANS Skills and Competencies and Most Prevalent SCANS Sub-Skills and Competencies 
Reflected in Each Industry’s Skill Standards. 
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These findings are supported by the results of a broad-based study identifying critical behaviors 
in a variety of occupations. The National Job Analysis Study (1995) had workers identify and 
rank work-related behaviors across a broad representative sample of occupations. Five of the ten 
most important behaviors (Listening to the concerns of clients/customers and responding. Ability 
to use a computer to locate, process, or communicate information, Providing information to 
people. Judging the importance, quality, and accuracy of information, and Listening to 
instructions from or concerns of supervisors or co-workers and responding) directly relate to the 
SCANS competencies and foundation skills identified as common to the four industries. 

Our findings also are supported by the Institute for Educational Leadership’s report. In Search of 
Commonalities (1996), describing identification of commonalities among twenty-two diverse 
industry skill standard sets. Similarly to IEL’s work, we culled common skills from standards 
sets including basic and occupational specific skills. The IEL study used the 0*NET skills 
framework as a basis for sorting common skills across industry skill standards. - 



Cross Industry Categories 

To create a workable set of cross-industry categories, given differences in foci among the four 
industries, we broke apart the very broad SCANS categories into areas of sub-skills that clustered 
together in relation to the standards of all four industries. We inductively generated eight 
categories that represent different areas of focus across the standards. These categories form a 
flexible framework, as we anticipate that some industries will focus more on some categories 
than on others. This will be important in designing and adapting assessment methods viable for 
different occupations and contexts. 

The eight cross-industry competency areas are defined as follows: 

• Client Orientation - Workers are sensitive to customer/client needs and adapt interactions, 
services, and products to best match these needs. New products, services, and markets are 
created as workers perceive the demand and innovate to best meet it. For example, 
electronics sales workers identify key decision makers in prospective companies, research 
company needs, and cross-check needs against service capabilities. 

(SCANS sub-skills: Serves Clients/Customers, Exercises Leadership, and Negotiates) 

• Teamwork - Workers support coworkers and collaborate with workers from different fields 
or industries. For example, human service workers may facilitate services and collaborate 
with client’s employer, school personnel, community service agencies, and health care 
agencies. 

(SCANS sub-skill: Participates as a Member of a Team) 

• Evaluating and Interpreting Information - Workers assess contexts and clients/customers and 
use this information to assist with decision making and goal setting. Feedback on the quality 
of industry services is used to improve quality and maintain a competitive edge. For 
example, human service workers fully assess clients’ physical, social, and mental well-being, 
as well as available supports, to assist in developing care and training plans to enhance client 
self-sufficiency. 
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(SCANS sub-skills: Acquires and Evaluates Information and Communicates and 
Interprets Information). 

• Awareness of Safety and Security Practices - Workers understand human, legal, and civil 
rights involved in appropriate practice in their field and work to protect the interests of their 
clients/customers or the company/organization. For example, health care workers need to 
understand and use safe work practices to prevent infections and injury to themselves and 
patients. 

(SCANS sub-skills: Understanding Systems; SCANS thinking skills and personal 
qualities: Decision Making, Problem Solving, and Integrity/Honesty). 

• Applying Technology (particularly computers) - Workers use technology across a variety of 
tasks both electronic and mechanical. For example, human service workers may need to be 
adept with a range of communication and adaptive equipment. 

(SCANS sub-skills: Uses Computers to Process Information, Selects Technology, and 
Applies Technology to Task). 

• Organizing Information - Workers research, develop, and maintain databases or files of 
information about the customers/clients and about available services, products, and resources. 
For example, electronics manufacturing workers collect and maintain information on 
production set-ups and processes to ensure continuity and to improve performance. 

(SCANS sub-skills: Acquires and Evaluates Information and Organizes and Maintains 
Information). 

• Managing Resources - Workers manage or assist clients with managing schedules, 
equipment, therapies or services, and materials. For example, retail workers may be 
responsible for taking inventory and reordering based on shopping patterns of their 
customers. 

(SCANS sub-skills: Allocates Time, Allocates Materials and Facility Resources). 

• Professional Orientation - Workers establish connections to a larger professional context. 
They complete necessary training, engage in continued professional development, and 
educate clients/customers and the community about relevant issues. For example, health care 
workers are better prepared for employment when they are familiar with various career 
alternatives in the industry and the types of preparation required for them. 

(SCANS sub-skills: Understands Systems; SCANS personal qualities: Responsibility, 
Self-Esteem, and Self-Management). 

These categories are certainly not comprehensive. Other common skill areas may exist among 
the industry standard sets and other standard sets may lead to the development of other or 
different categories. In addition, our cross-industry standards are not fully validated standards, 
having been reviewed only by project partners and staff. 
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Preliminary Assessment Development Efforts 



We selected two performance-based assessment methods on which to focus our assessment 
development efforts: scenarios and portfolios. Scenarios and portfolios offer contrasting but 
complementary assessment methods that may be useful for large-scale or high-stakes assessment 
purposes. In general, performance-based assessments also allow for “stronger links among 
assessments, standards, and instructional/training goals” (Rabinowitz, 1996). They demand 
direct evidence of the ability to actually perform integrated, complex skills; they can be 
embedded in workplace contexts; and they often have high credibility. For the purposes of 
illuminating the connection between the cross-industry standards and assessment development, 
this paper will describe one type of assessment method focused on by the NSSAC project, 
written scenarios, and present example scenarios developed during NSSAC piloting efforts. 

The written scenario involves presenting a work-related situation for which the respondent is 
asked to develop and present a solution. Scenarios challenge respondents to apply their 
workplace knowledge and use their critical thinking skills. Real world problems are often ill- 
defined and require workers to integrate different kinds of knowledge, then apply that knowledge 
in a new context. Students or workers must use critical thinking skills, such as organizing, 
summarizing, classifying, comparing, generalizing, and inferring, to construct effective 
solutions. Scenarios are on-demand assessments, administered under fairly standardized 
conditions during a short-time period (40-45 minutes). Additionally, scenario formats are 
already being used in industry for a variety of purposes. For instance, the Bioscience Standards 
by EDC used a scenario format for presenting their skill standards. 

We began the assessment development process by asking each project partner to rank the eight 
cross-industry standards according to their 1) importance to the industry and 2) applicability for 
written scenario assessments. Scenario development then focused on the top-ranked cross- 
industry standards. Notably, Client Orientation and Teamwork were selected as the most relevant 
cross-industry standard categories and most amenable to scenario assessments. This is not 
surprising. At least one industry, retail, independently surveyed its constituents finding that 
nearly 90 percent of retail companies provide training for employees in customer service 
methods (Troppe, 1996). 



Developing Scenario Prototypes 

For assessments tasks that are embedded in the workplace and collected on an ongoing basis, one 
of the keys is to find ways to develop, monitor and assure the consistency and comparability of 
judgments across widespread sites or parts of the system (Wirt, 1993). One strategy that can be 
used to make assessments more comparable to each other involves using an assessment 
prototype or template to guide development. 

A prototype or template serves the purpose of ensuring that an assessment task contains all of the 
necessary information presented in an identical way. For examinees, using such a template 
means they can gain familiarity with the process of responding to such a task and know what to 
do in a potentially stressful testing situation. For the assessment developer, the prototype 
provides a template for development and review of tasks. In these situations, valuable time and 



template for development and review of tasks. In these situations, valuable time and resources 
are not lost reinventing the format; instead, time can be devoted to developing appropriate, 
specific content. 

A prototype can be format guidelines or it can be more developed and linked to a specific kind of 
task, problem, or situation. One can adapt such a prototype to multiple industry contexts, 
situations and settings. 

Using a scenario format piloted extensively with the Career-Technical Assessment Program (C- 
TAP) in the state of California, we adapted this template to create cross industry written 
scenarios. This template includes four scenario components: a title, prompt, instructions, and 
evaluation criteria. Our first example scenario shows these four components in relation to an 
actual cross-industry scenario. 

As we worked with our industry partners to develop written scenarios, we focused on the generic 
types of work-related problem solving situations that evolved. This became an informal typology 
of work-related problem solving tasks that functioned across industry contexts. We identified 
five different prototypes of problem solving tasks which are described in the following section. 
Not all of these types of problems apply across every industry. Our five problem situations 
include: 

• means-end scenarios, 

• crisis and follow-up scenarios, 

• roles and responsibilities scenarios, 

• developing recommendations scenarios, and 

• competing clients or priorities scenarios. 

Each type of problem solving situation will be described in turn. 



Means-End Scenarios 

Means-end scenarios require the use of general problem solving strategies in a skill area related 
to a specific job. Respondents are given a hypothetical situation in which they are trying to reach 
some goal and need to find a means for getting there. Instructions for the scenario usually prime 
the respondent to use means-end analysis to develop a solution by breaking the end goal into sub 
goals that needed to be discussed in the answer as well. This type of scenario is broadly 
applicable. All NSS AC industry partners developed and piloted means-end scenarios. 

Figure 4 presents an example means-end scenario. This scenario. Display Dilemma, is a retail 
scenario which addresses one module of the NRF Skill Standards, setting up and monitoring 
displays. Specifically, the module is used as the basis for assessing a sales associate’s skills in 
tailoring displays to teenage customers. In writing a response, a respondent must draw on prior 
knowledge to design a display geared to teenagers, think of ways to judge the display’s 
effectiveness and plan how different information gained through this process would be used to 
improve the display. Each of these steps is specified as a point to be addressed in the response. 
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Figure 4. Example of a Means-end Scenario 



Title 



Display Dilemma (A Client Orientation Scenario in Retail) 



Prompt 



Scenario 



You are a sales associate in a large department store. Your store is having its annual weekend sale and you are 
expecting a large number of teenage customers. As you arrive for work you notice your department’s sale was not 
constructed the previous night. Your supervisor asks you to spend the morning setting up the display. 



Instructions 



Instructions 



Think about what you know about presentation and displays. Describe in detail what you need to consider as you set 
up the display and explain your ideas. Also, describe several ways to monitor the effectiveness of the display during 
the sale. Discuss the information each monitoring activity would provide and what you could do to improve the 
display. 



Evaluation Criteria 



To receive a Proficient rating on this task, you must show all of the following: 



1. Knowledge of: 

• monitoring effectiveness of displays 

• arranging merchandise to maximize sales 



2. Ability to propose an effective solution to this scenario 

3. Ability to communicate clearly in writing 



At each step different issues may need to be considered. For example, a respondent might say 
that color, style, and placards or signs should be considered during display design. For the 
second step, monitoring effectiveness, customer behavior, that is, whether or not customers stop 
to look at the display, as well as daily sales totals could be checked. Finally, checking before and 
after sales for an item could inform designing future displays. 

Crisis and Follow-up Scenarios 

Crisis and follow-up scenarios present a critical situation typical in each industry that needs to be 
resolved immediately. Instructions in the scenario ask the respondent to tell how they would deal 
with the crisis and then what long-term strategies they would use to maintain good practice. For 
NSSAC, three out of the four participating industries developed and piloted this type of scenario; 
Electronics, Retail, and Health. We might expect that most companies have crisis situations arise 
that do involve long-term follow-up, therefore, this type of scenario is broadly applicable. 

Our example of a crisis scenario is A Change Order , a teamwork scenario which was developed 
and piloted with the electronics industry. This scenario, shown in Figure 5 presents the 
respondent with the problem of receiving a manufacturing change on an important order on a 
short deadline. The situation involves solving this immediate problem, as well as telling what 
steps you would take to maintain positive team spirit and group cohesiveness. Instructions to this 
type of scenario ask the respondent to address both the short-term and long-term aspects of the 
problem in presenting a solution. 
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Figure 5. Example of a Crisis and Follow-up Scenario 



Title 


A Change Order (A Teamwork Scenario in Electronics ) 


Prompt 


Scenario 




You are a team leader for a computer hardware manufacturing unit. You have just received a change order on a 
motherboard that your unit is fabricating. You know this motherboard is part of a large order for a customer and 
needs to be shipped overseas before 5:00 PM today. 


Instructions 


Instructions 




Think about what you know about initiating rapid change in the manufacturing process and building team 
support. Describe the steps you would take to identify the required changes and carry out the new 
specifications. Also, describe how you would maintain team spirit and cohesiveness during this process. 


Evaluation Criteria 


To receive a Proficient rating on this task, you must show all of the following: 




1. Knowledge of: 

• change processes in manufacturing 

• leading work teams 




2. Ability to propose an effective solution to this scenario 




3. Ability to communicate clearly in writing 



Roles and Responsibilities 

This type of scenario focuses on situations in which respondents are requested to delineate the roles 
and responsibilities of different players in the workplace. The scenarios often involve knowledge of 
the scope of practice for specific jobs and how that relates to accomplishment of a larger task. Some 
may relate to line of authority situations, such as supervisor’s responsibility for accurate work and 
task completion. These types of problem situations, although common, may not be universal among 
industries. For example only two NSSAC industries, Electronics and Health, developed and piloted 
roles and responsibilities scenarios. 

The health care scenario, shown in Figure 6, The Working Supervisor, presents a situation in 
which the respondent’s nursing supervisor is busy taking orders from a physician. Consequently, 
the supervisor has been unable to attend to patient care. When one of the supervisor’s patients 
calls for immediate attention, the respondent must develop a solution that manages the situation, 
ensuring that proper procedures are followed and patient needs are met. 
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Figure 6. Example of a Roles and Responsibilities Scenario 



Title 


The Working Supervisor (A Teamwork Scenario in Health Care ) 


Prompt 


Scenario 




You are a patient care technician. Your supervisor, a charge nurse, is assigned patients in addition to her other duties. 
This shift, she has spent a lot of time on the phone with a doctor dealing with necessary changes to patient’s orders. 
While your supervisor is busy on the phone, one of her patients has turned on the call light. You observe that your 
supervisor is not responding to the patient’s call light. 


Instructions 


Instructions 




Think about what you know about working as a member of a healthcare team and the roles and responsibilities of 
each team member. Given your scope of practice as a PCT, explain how you would ensure that appropriate 
patient care is given in this situation. Give examples of the kind of patient care you could administer on your 
own, could administer with approval, or could not administer (some one else would need to do ) in this situation. 


Evaluation Criteria 


To receive a Proficient rating on this task, yon must show all of the following: 




1. Knowledge of: 

• scope of practice 

• teamwork skills 




2. Ability to propose an effective solution to this scenario 




3. Ability to communicate clearly in writing 



In this situation, the respondent may be able to assist by providing some, but not all, of the same 
functions as the supervisor. The respondent must demonstrate knowledge of appropriate patient 
care and what they could or could not do, as well as how they might best contribute to the team 
effort. For instance, a respondent might say he or she would let the supervisor know they are 
answering the call light, make the patient comfortable without administering medication (citing 
specific activities to accomplish this) and/or find an individual who can administer medication or 
other therapies, as needed. 



Developing Recommendations Scenarios 

Developing recommendations scenarios present a situation in which the respondent is required to 
assess the extent or nature of a problem, draw on their knowledge of supports related to the 
problem and then compile a list of alternatives or options for the client customer to follow. In 
this type of scenario the options are different recommendations for actions or resources and the 
respondent is usually asked to provide a reason or justification for each option. Although only 
two NSSAC industries used this type of scenario, human services and retail, providing advice 
based on evaluation of a problem can be expected to be a universal and, thus, cross-industry type 
dilemma. 

Our example of a recommendations scenario is Supporting Participant Self-Advocacy, a human 
services scenario (see Figure 7). This scenario involves a community support service worker 
who needs to help a client procure services for bathroom modification. Complications include 
prior classification of the housing unit and the client being a person with a disability. The 
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respondent must develop a solution that includes several recommendations for strategies the 
client can use to get assistance. In pilot testing, respondents suggested strategies such as getting a 
copy of the lease or rental agreement, contacting outside agencies, and/or moving to a new 
apartment. 

Figure 7. Example of a Developing Recommendations Scenario 



Title 


Supporting Participant Self-Advocacy (A Client Orientation Scenario in Human Services) 


Prompt 


Scenario 




You work as a personal assistance practitioner providing support to people with disabilities who live independently. John, a 
new participant receiving support services, told you during an initial conversation that the bathroom facilities in his 
condominium were not suitably adapted to his needs. It was difficult for John to use the bathroom under these conditions. 
Because his housing arrangement had been classified as “accessible,” John wanted to ask the condominium management for 
additional modifications. 


Instructions 


Instructions 




Think about what you know about supporting participants with disabilities in obtaining assistance from state, community, 
and health support agencies. Describe in detail what you should do and say so that John fully understands what options are 
available for getting assistance and what strategies he might use for obtaining aid to get his bathroom facilities 
appropriately adapted. 


Evaluation Criteria 


To receive a Proficient rating on this task, you must show all of the following: 




1. Knowledge of: 

• self-advocacy methods and support groups 

• how to increase awareness of self-advocacy methods and groups 

• adapting interactions to best match participant needs 




2. Ability to propose an effective solution to this scenario 




3. Ability to communicate clearly in writing 



Competing Clients or Priorities Scenarios 

The competing clients or priorities scenario presents the equivalent of an in-basket task for line 
workers. Developing and presenting a solution involves juggling conflicting priorities .along 
more than one dimension. The respondent has to prioritize competing tasks, keeping their clients 
in mind, and then explain how they would communicate their decision to the clients Two of the 
NSSAC industries developed and piloted these type of scenarios, health care and electronics. In 
contrast to the health care example shown above, the electronics scenario involved different 
types of services among internal versus external users. 

The competing clients or priorities scenario in Figure 8, Prioritizing Patients ’ Needs , presents 
the situation of a health care technician who is responsible for several patients with different 
kinds of needs and different levels of criticality. Clinical or best practice considerations, as well 
as patient interaction considerations are involved. A newly arrived patient requires that 
specimens for lab tests be collected that will require personal attention for at least a half hour. 
Meanwhile, another patient is angry and demanding attention. The respondent must determine 
which patient and tasks need to be attended to first and then communicate their decision to their 
patients. Many respondents develop solutions that involve settling the angry patient down by 
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explaining why they were delayed, bringing him juice or a snack, and then explaining to him that 
another patient needs their attention. Respondents state that they plan to return, after collecting 
the required lab samples, to perform their regular duties and other types of patient care. 

Figure 8. Example of a Competing Clients or Priorities Scenario 



Title Prioritizing Patients' Needs (A Client Orientation Scenario in Health Care) 

Prompt 

Scenario 



You are a patient care technician in a busy Intensive Care Unit (ICU) for an acute care facility. Today, you have four 
patients. This includes a new patients, just admitted to the unit. The new patient, Ms. Turrell, needs to have urine and 
blood specimens collected as soon as possible. Mr. Jones, another patient, has had his call light on for the last five 
minutes. So far you have been unable to answer it. Mr. Jones is now angrily yelling that he wants his dinner. 



Instructions 



Instructions 

Think about what you know about communication and patient care. Describe in detail what you should do so that patient care activities 
are prioritized in an appropnate manner. Explain what you would do or say to make sure Mr. Jones’s needs are recognized 



Evaluation Criteria 



To receive a Proficient rating on this task, you must show all of the following: 

1 . Knowledge of client interaction including: 

• providing personal care for patients 

* communicating effectively 

2. Ability to propose an effective solution to this scenario 



3. Ability to communicate clearly in writing 



Using the Written Scenario Prototypes for Assessment Development 

The problem situations represented in our five prototypes provide a range of cognitive complex 
tasks to challenge respondents to apply general as well as work-related knowledge and skills. 

The cross industry written scenario prototypes can be used as templates for constructing 
industry-specific contexts for such scenarios. Interestingly, development and implementation of 
the scenarios raised issues related to scope of practice and level of specificity levels. Skill 
standards reflect the needs of an entire industry, and each store, region, or sector of the industry 
has their own policies that might differ from the national skill standards or from store to store or 
area to area. How these issues played out in relation to scenario development is described below. 

• Scope of practice - Industries — and positions within them — are structured quite differendy. 
Industry skill standards reflect this difference. For instance, the retail skill standards are written 
for a specific job — professional sales associate — whereas the health core skill standards feature 
both core and occupational cluster standards. Although the health care scenarios were written to 
the core skill standards (e.g., teamwork), a specific job/position was used to contextualize the 
scenario prompt. Specifically, pilot-test health care scenarios placed the respondent in the role of a 
patient care technician, a newer multi-skilled worker designation. There is a trend in the industry 
in some regions to support and preferentially employ these types of workers. However, those pilot 
sites in areas where more traditional types of worker designations were the norm had to pre-assess 
the scenario tasks to decide if their program or training (e.g., Certified Nursing Assistant) related 
to the same level of performance expectations. For example, although a patient care technician 
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and a certified nursing assistant both perform overlapping tasks, this is not true of all tasks for 
which each type of worker is responsible. 

Although scope of practice issues were most evident in the health care industry, they were 
evident in other industries as well. The retail scenarios posed some issues for respondents as 
well. Some respondents said they would not be called upon to establish displays or that their 
place of employment did not set up displays, such as the one posed in the scenario. So, even 
though the retail standards are written for a single position, there was still a scope of practice 
issue within the scenarios written to target this set of standards. 

» Level of generality vs. specificity - Development efforts focused on providing specific situations 
yet making them generic enough to cut across the industry. This posed some concerns; some 
respondents mentioned they needed more information before responding (e.g., human services), 
while others commented that the topic (e.g., electronics) was too narrow. This is to be expected 
when developing for large industries. For example, in human services some community support 
service workers work with mentally retarded (MR) patients while others work in child welfare 
environments. Developing scenarios that were amenable across these different settings, yet 
provided enough detail for a realistic context was challenging. 

One possible strategy to address the generality vs. specificity tension is to allow respondents 
to tailor their scenario responses to the exact context within which they operate. For example, 
a Human Services scenario might describe a self-advocacy situation. The respondent might 
be instructed to select the type of patient he or she works with and then talk about 
appropriate steps to take in relation to this targeted population. The challenge here, of course, 
is developing scenarios that allow for these type of options yet still are comparable across 
employment situations. 

Additionally, we found that project participants early on in the collaborative effort expressed 
their strong preference for couching assessment tasks in the context of their particular industry. 
The reasons given for this preference related to both content validity and face validity 
considerations. A major content validity consideration was that knowledge and skills are 
inextricably bound to context. For example, participants felt that although an emphasis on 
teamwork skills is common across industries, the specific teamwork skills differ across 
industries. Similarly, a face validity consideration is that in order for an assessment task to be 
* credible with the stakeholders in a particular industry, that assessment tool must look like it is 
assessing skills specific to that industry. In fact, even within the same industry or occupational 
concentration it often proved difficult to develop scenarios that were considered by our content 
experts to have content and face validity for more than one specific occupational area. 

Despite these issues, the NSSC project documented that there was widespread support among 
both educators and employers for written scenarios and for the idea of core cross-industry 
standard areas. Stakeholders recognize that written scenarios require respondents to apply 
occupationally-related knowledge to solve realistic problems. Both employers and educators 
like the written scenario because they represent a relatively cost-efficient means of measuring 
competency in relation to targeted standards. The scenario approach affords great flexibility; 
prompts and responses can be adapted to assessment modes other than written, such as video or 
oral assessments. By offering common processes and templates to stakeholders, we can help 
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“demystify” assessment for the field and help build recognition and support for skill standards 
and related assessment. 
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